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1. Introduction 


Programs for promoting health behaviors have been developed for 
adoption in various settings such as schools (Hoelscher et al., 2010; 
Sallis et al., 1997), afterschool programs (Beets et al., 2009), child care 
centers (Ward et al., 2010), community health and mental health clinics 
(Aarons et al., 2011), and other settings (Soler et al., 2010). While many 
of these programs have established efficacy, implementation-related 
challenges often limit their impact in real-world applications (Glasgow 
et al., 2003; Glasgow and Emmons, 2007). As a result, an emerging 
body of literature has pointed to the need to better consider im- 
plementation factors related to real-world applications when devel- 
oping setting-based health promotion programs (Tomoaia-Cotisel et al., 
2013; Damschroder et al., 2009). Also of critical importance is to 
prioritize the dissemination of programs that address implementation 
factors, and to supplement programs that do not address such factors 
with appropriate and effective supports, to maximize implementation 
rates and quality. 

Implementation factors are the elements that surround a particular 
intervention, which impact the extent to which it is put into practice as 
intended, and the extent to which it is successful in promoting the in- 
tended outcomes (Damschroder et al., 2009; Rabin et al., 2008). These 
factors span multiple levels of influence, in accordance with ecological 
models, including individual-, interpersonal-, and environmental-level 
influences (Sallis et al., 2015). A growing body of evidence shows the 
importance of such factors to ensure “real-world” program effective- 
ness-even the most efficacious programs will not work without quality 
implementation (Tomoaia-Cotisel et al., 2013; Durlak and DuPre, 
2008). Several implementation science frameworks have been devel- 
oped to organize the conceptualization of implementation factors 
(Tabak et al., 2012a; Nilsen, 2015). Broadly, these frameworks include 
characteristics of the intervention, social and physical environment, 


availability of resources, and characteristics of the individuals involved 
in implementation. While implementation science frameworks have 
traditionally been used to identify barriers and facilitators to im- 
plementation and/or guide implementation efforts (Nilsen, 2015; Tabak 
et al., 2012b), they should also be used to inform the selection, prior- 
itization, refinement, and use of available programs or program com- 
ponents to maximize implementation rates, quality, and effectiveness. 

Tools and methods have been developed and applied to rate po- 
licies, practices, and programs on characteristics such as their quality 
and effectiveness (Schwartz et al., 2009; Collins et al., 2002; Carlson 
et al., 2013). However, tools for assessing implementation factors 
within available health promotion programs have been given little at- 
tention. Therefore, the purpose of the present study was to develop and 
test a tool and methodology for assessing coverage of a wide range of 
theoretically-based implementation factors within available health 
promotion programs. The study was informed by the Consolidated 
Framework for Implementation Research (CFIR) (Damschroder et al., 
2009), which is a consolidation of multiple frameworks and has been 
used across a variety of studies (Kirk et al., 2016). The tool involves a 
coding system that allows researchers and other stakeholders to review 
and select programs to support their mission of health promotion, with 
key consideration of each program's potential for optimal im- 
plementation, as a result of the types of implementation supports that 
accompany each program. This paper describes the process of devel- 
oping and testing a tool specific to classroom-based physical activity 
(CBPA) programs, with methodology that can be applied to a wide 
range of health promotion programs, particularly those that are setting- 
or organization-based. 
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2. Methods 
2.1. Study overview 


This study involved the development and testing of a coding tool 
and methodology for assessing whether and to what extent available 
CBPA programs address factors that relate to improved implementation 
in “real-world” settings. To accomplish this, a team of experts with 
experience in CBPA research was organized. The team selected an im- 
plementation science framework to guide the study, developed a set of 
codes specific to the implementation factors, developed code definitions 
and a coding guide, systematically identified and coded available CBPA 
programs, tested the quality of the codes by having two coders in- 
dependently code each program, and subsequently calculated inter- 
rater reliability. 


2.2. Selection of implementation framework 


CFIR was selected to guide the development of the implementation 
factor assessment tool because of its comprehensiveness (Damschroder 
et al., 2009) relevance to implementation effectiveness in previous 
studies (Kirk et al., 2016), and coverage of implementation factors 
acting at multiple ecological levels of influence. CFIR includes 36 
constructs organized under five domains: Intervention Characteristics 
(key attributes of interventions influence the success of implementa- 
tion), Outer Setting (factors outside of an organization that affect im- 
plementation), Inner Setting (factors within an organization that affect 
implementation), Characteristics of Individuals (perceptions, attitudes, 
and motivation of individuals within the organization), and Process 
(strategies used to support implementation) (Damschroder et al., 2009). 
Other frameworks (Tabak et al., 2012a) were considered and would 
have been appropriate (Atkins et al., 2017; Kitson et al., 2008; Aarons 
et al., 2011). Ultimately, CFIR was chosen over these frameworks be- 
cause of the team's familiarity with the framework and the extensive 
resources and descriptions available on the CFIR website (Consolidated 
Framework for Implementation Research, 2014). 


2.3. Development of codes and code definitions 


The CFIR construct list was used to facilitate the development of the 
list of CBPA-specific implementation factor “codes” (e.g., Goal Setting) 
and code definitions. The focus was on implementation rather than 
adoption because implementation is an ongoing effort that is more 
closely linked with effectiveness than adoption (Rabin et al., 2008). A 
brief review of the literature was conducted to understand published 
barriers and facilitators to CBPA implementation. An iterative approach 
involving all authors was used to develop the codes and corresponding 
definitions and match them with the CFIR constructs. The primary in- 
vestigator and a research assistant first drafted the initial code names 
and definitions, mapped to CFIR. Each CBPA-specific code, definition, 
and related CFIR construct were then discussed during a series of 
meetings with the other study team members. During and after each 
call, the code names, definitions, and mapping to CFIR were refined, 
and codes were added and removed. After the fourth meeting, con- 
sensus was reached and the code list was finalized. 

The code development process aimed to produce a tool with code 
names and definitions that aligned to CFIR, but were more narrowly 
focused to the school setting and to CBPA programs. Specific differ- 
entiating criteria were used to facilitate coding decisions and to max- 
imize relevance to CBPA and minimize subjectivity when applying the 
codes. 

It was acknowledged that various levels of influence affect im- 
plementation, and because many CBPA program materials were be- 
lieved to focus more on the teacher-student relationship (e.g., influen- 
cing student behavior change) and less on the school-teacher 
relationship (e.g., influencing teacher behavior change), each of the 
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agreed upon CBPA-specific codes was categorized in this way. This 
grouping excluded the CFIR Intervention Characteristics codes because 
they were not specific to either relationship. 


2.4. Identifying programs 


A systematic search of both scientific and grey literature was con- 
ducted in 2017 to identify programs directed at delivering physical 
activity in the classroom. It was important to search outside of scientific 
search engines because health promotion programs are often developed 
and/or disseminated by community organizations, health organiza- 
tions, and businesses. Such programs may not have been research- 
tested but were often research-based, and in some cases (e.g., with re- 
gards to CBPA) may be more likely to be adopted than researcher-de- 
veloped programs. Search terms included classroom physical activity + 
programs + interventions; active classroom; activity breaks; active 
lessons; classroom breaks + exercise; classroom physical activity. 
Search engines included PubMed and Google Scholar for scientific 
publications, and Google for grey literature. Using Google, the first 50 
data hits were examined for each search term. In addition, reference 
lists of identified scientific publications and programs materials were 
reviewed for potentially relevant programs or studies. Most programs 
had their own website but some were housed on another website such 
as one hosted by a department of education, health department, or 
health organization. 

Potential programs were screened for inclusion as they were iden- 
tified. Inclusion criteria were: published in English language; targeted 
increasing physical activity in the classroom (but could also address 
other sources of in-school physical activity [e.g., Physical Education]); 
and targeted any/all of grades K-6th. Exclusion criteria for programs 
were: exclusively offered content designed only for physical education 
class, recess or afterschool programs, not classroom physical activity; 
created as part of a research effort but were not publicly available even 
upon request; previously available, but no longer had active websites or 
links; and were based outside of the United States. 


2.5. Creating directory of program materials 


One staff member compiled all materials for each program. First, 
each program's webpage was visited and thoroughly searched. 
Downloadable materials (e.g., PDFs, PowerPoints, manuals/guides, 
handouts, posters) were obtained and compiled in an electronic library. 
In circumstances when the website mentioned that trainings, materials, 
or resources were available but were not posted on the website, the staff 
member contacted the program directly to request these materials. In 
some cases, website user accounts were created or materials were 
purchased. A summary document was created for each program, which 
included an overview of the program and details about its materials, 
and notes regarding materials that were available through direct con- 
tact with the program or on the program's website. 


2.6. Coding procedures 


A coding guide was created to facilitate the coding of each CBPA 
program. The coding guide included the CBPA-specific code names and 
definitions, as well as the related CFIR construct name and short de- 
scription. Coders were instructed to review all program materials 
compiled in the library and thoroughly search and review the program's 
website to obtain information potentially missed by the staff member 
who initially compiled the resources. Coders were also instructed to 
exclude program content related to something other than CBPA. This 
was relevant because several programs addressed physical activity and/ 
or nutrition during other aspects of the school day (e.g., during Physical 
Education, after school) in addition to CBPA. The team tried to consider 
how a school or teacher would access/use the resources if they wanted 
to only implement the CBPA portion of the program. 
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The CBPA-specific codes within the Intervention Characteristics 
domain were objectively based, relatively easily identifiable, and the 
coding required little-to-no judgment or interpretation from the coder. 
A score option of 0 = no vs. 1 = yes was used for these codes and they 
were coded by a single coder because double coding was not expected 
to improve the accuracy of the scores. All other CBPA-specific codes 
were coded as: 0 = not covered in the program, 1 = covered in the 
program but at a minimal level, and 2 = covered more than minimally 
in the program. For these codes, two coders independently coded each 
program and discrepancies were reconciled through discussion among 
the two coders. In general, 2's were only used to identify and reflect 
programs that thoroughly addressed a given concept. 

For training, all coders independently coded the same three pro- 
grams, one at a time, with discussion after each focused on improving 
inter-rater reliability for the subsequent program coding. A pool of five 
coders was used. One coder coded all Intervention Characteristics, with 
all other domains double coded by 2 of the 5 coders (with an ap- 
proximately equal number of programs being coded by each). Coders 
were a mix of Masters-level staff and students, PhD students, and 
postdoctoral fellows with training in research and health promotion. 


2.7. Statistical analyses 


Inter-rater agreement was assessed using intraclass correlation 
coefficients (ICCs) and percent agreement, for those coded twice. ICCs 
were calculated for the 0-2 scores using one-way random effects models 
for the average of the measures. Percent agreement was calculated for 
the 0-2 scores. In addition, because coding of “2” was very rare (only 
6.7% of codes received any score of 2, across all programs), additional 
analyses were conducted with codes of 1 and 2 collapsed to 1, versus 0. 
This dichotomization allowed for examination of whether inter-rater 
reliability was improved when considering “any versus none” regarding 
the inclusion of each implementation factor, by grouping the minimally 
(Durlak and DuPre, 2008) and more than minimally (Carlson et al., 
2013) covered scores. ICC magnitude was classified using criteria of 
poor (<0.40), fair (0.41-0.60), good (0.61-0.80), and excellent 
(0.81-1.0) (Cicchetti, 1994) and percent agreement was evaluated with 
the criteria of =75% as good to excellent, 60-74% as moderate, 
and < 60% as poor (Landis and Koch, 1977). Both metrics were con- 
sidered while interpreting inter-rater reliability, as ICC values can fal- 
sely suggest poor agreement when there is little variability in responses 
across programs. To summarize the responses within the CFIR domains 
of Inner Setting, Characteristics of Individuals, and Process, and the 
groupings by the school-teacher, and teacher-student relationship, 
“index scores,” or sum scores within the domains, were calculated. 
Inter-rater reliability of the index scores was tested using one-way 
random effects ICCs. Analyses were conducted in SPSS version 23. 


3. Results 
3.1. Number of programs identified and final list of codes 


A total of 37 programs were identified through the search process. 
The final coding guide included 51 CBPA-specific codes. The code 
names, code definitions, and mappings to CFIR are presented in the 
Appendix. Some CFIR constructs were mapped to multiple CBPA-spe- 
cific codes, whereas others were not mapped to any CBPA-specific 
constructs, based on the study team's perception of the importance and 
relevance of the construct to CBPA. This process resulted in 23 CBPA- 
specific codes for the Intervention Characteristics domain of CFIR, 
which were coded 0-1 by a single coder, and the other three domains 
that were coded 0-2 by two coders, including 14 codes for the Inner 
Setting, 6 for the Characteristics of Individuals, and 8 for Process. The 
Outer Setting domain was not covered because the codes developed for 
this domain overlapped with other domains and constructs (e.g., the 
External Change Agents Construct was included instead in the Process 
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Table 1 
Rater agreement for implementation factors coded by two coders (N = 37 
programs). 


Construct ICC for 3- % agreement % agreement for 
level score for 3-level score dichotomized score 

Inner setting 

Communication* 0.630 81.1% 83.8% 

Policy incorporation® — 0.023 64.9% 63.9% 

Marketing materials 0.135 56.8% 62.2% 
teachers* 

Marketing materials 0.650 73.0% 86.5% 
students/parents” 

Gauging/affecting 0.854 89.2% 94.6% 
climate* 

Leadership initial buy in* 0.488 70.3% 75.7% 

Student management” 0.729 78.4% 86.5% 

Compatibility 0.539 56.7% 70.3% 
adaptations” 

Incentives* —0.130 67.6% 67.6% 

Goal setting" 0.242 86.5% 86.5% 

Monitoring* 0.679 75.0% 75.7% 

School readiness* —0.091 89.2% 89.2% 

Leadership engagement 0.366 86.5% 86.5% 
post adoption* 

Classroom structure” 0.396 62.2% 70.3% 

Characteristics of individuals 

Health benefits* 0.637 51.3% 67.6% 

Non-health benefits* 0.695 64.8% 89.2% 

Teacher motivation/ 0.036 75.7% 75.7% 
attitudes around 
program* 

Self-efficacy* 0.100 75.7% 75.7% 

Teacher stage of change* — 0.029 94.6% 94.6% 

Teacher attitude/value 0.088 78.4% 78.4% 
toward PA* 

Process 

Scheduling materials? 0.822 73.0% 81.1% 

Dose/dose quantity” 0.775 72.9% 86.5% 

Teacher participation” 0.796 75.7% 81.1% 

Implementation leaders* 0.615 73.0% 73.0% 

External involvement* 0.467 73.0% 78.4% 

External Information 0.409 56.8% 64.9% 
sharing* 

Accountability* 0.795 97.3% 97.3% 

Outcomes* 0.704 73.0% 81.1% 


* Construct affects school-teacher relationship for implementation (21 con- 
structs). 

> Construct affects teacher-student relationship for implementation (7 con- 
structs). 


domain), so these codes were mapped to these other constructs. The 
school-teacher relationship was represented in 21 codes, and the tea- 
cher-student relationship in the remaining 7 codes. Across the 28 codes 
that were scored on the 0-2 scale, 6.7% were scored as 2. Across the 
CFIR domains, several CFIR constructs were not covered (CFIR 
Research Team, 2015), either because they were thought to overlap 
with other constructs, they were considered not be to relevant to CPBA 
programs, or they were not expected to differ across programs. 


3.2. Inter-rater agreement for CBPA codes 


Inter-rater agreement results (prior to reconciliation) are detailed in 
Table 1. Fourteen of the 28 codes had excellent inter-rater reliability 
based on an ICC > 0.80 and/or a percent agreement =75%. Six codes 
had good reliability based on an ICC between 0.61 and 0.80. Five codes 
had moderate inter-rater reliability based on a percent agreement be- 
tween 60 and 74.9% and an ICC between 0.41 and 0.60. The remaining 
three codes had poor agreement. Percent agreement for these codes was 
57% and ICCs were fair (0.41-0.60) or poor (<0.40). After all codes 
were dichotomized as 0 vs. 1, 20 of the 28 codes had good to excellent 
percent agreements (above 75%) and the remaining 8 had moderate 
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Table 2 

Inter-rater reliability for index scores. 
Index Icc* cc? 
Inner setting (14 codes) 0.606 0.592 
Characteristics of individuals (6 codes) 0.694 0.614 
Process (8 codes) 0.804 0.765 
School-teacher relationship (21 codes) 0.681 0.638 
Teacher-student relationship (7 codes) 0.847 0.877 


* For index scores that comprise a sum of the 3-level scores for each code. 
> For index scores that comprise a sum of the dichotomized scores for each 
code. 


percent agreements (60-74.9%). 


3.3. Inter-rater reliability for index scores 


Table 2 presents results of the inter-rater reliability analyses per- 
taining to the multi-code indices. ICCs for the indices calculated from 
the codes scored 0-2 were all good or excellent, ranging from 0.606 to 
0.847. ICCs for the four of the five indices calculated from the codes 
scored 0-1 were good or excellent, ranging from 0.614 to 0.877. The 
ICC for the remaining index (Inner Setting) was fair (ICC = 0.592). 


4. Discussion 


This study presents the process of developing and testing a theore- 
tically-based tool and methodology to assess coverage of implementa- 
tion factors in setting- and organization-based health promotion pro- 
grams that have already been developed and disseminated for use. 
Existing programs often include intervention strategies and content 
focused on delivery of the intervention to the end target (e.g., deli- 
vering CBPA to students). However, little is known about how these 
programs address implementation factors to integrate the intervention 
within the organization and to target behavior change in those who will 
deliver and support the intervention. A CFIR-based tool was created 
that can be adapted to assess a variety of health promotion programs 
targeting a variety of settings such as schools, child care, health ser- 
vices, and worksites. The inter-rater reliability results indicate that the 
code development process and coding process resulted in code defini- 
tions and coder scores that were of acceptable quality. This tool and 
methodology appear promising for identifying programs to prioritize 
for dissemination, guiding the selection of programs or program content 
to fit the needs of individual contexts, and identifying gaps across 
programs to support efforts to develop and test supplemental im- 
plementation supports. 

Fifty-one CBPA-specific codes were developed to cover im- 
plementation factors from CFIR. The 23 codes for the Intervention 
Characteristics domain of CFIR were relatively straightforward to 
identify and assess. These codes generally captured the intervention 
options provided by the program and are likely considered critical 
factors when considering whether to adopt a program and which pro- 
gram to adopt. These characteristics transcend health promotion pro- 
grams, and can be applied to a variety of interventions. A library or 
repository that provides information on these characteristics across 
available programs would provide a valuable resource from which end 
users could choose a program that aligns with their needs. As is known 
from various implementation theories including CFIR, information 
pertaining to Intervention Characteristics is necessary, but insufficient, 
for organizations to successfully implement programs. The 28 codes for 
the Inner Setting, Characteristics of Individuals, and Process domains of 
CFIR generally covered additional factors related to integrating the 
intervention within the organization, and promoting behavior change 
among those who have a role in intervention delivery. Implementation 
theory indicates that when this additional information is provided, 
programs are more successfully adopted, implemented, and sustained 
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(Nilsen, 2015). However, the current study found that such information 
was often not included in existing programs for promoting CBPA. Thus, 
researchers and program developers should include and continue to test 
the value of reporting this information in programs. 

In this study, some codes were challenging to use because they re- 
quired reviewers to conduct detailed reviews of program materials and 
some subjectivity and judgment was required of the coder. Using a 0-2 
scale (not covered, covered minimally, covered more than minimally) 
was advantageous because it allowed for the identification of content 
that addressed the code/factor more thoroughly, as minimal coverage is 
not likely to be of great benefit to supporting implementation. 
However, inter-rater reliability was less than ideal for some codes when 
using the 0-2 scale, suggesting that it was difficult for coders to identify 
the threshold used to represent more than minimal coverage. All codes 
had acceptable inter-rater reliability for the 0-1 scale, so this scale is 
preferred and the 0-2 scale should be used with caution. 

Test-retest reliability values were acceptable for all indices. The 
advantage of the indices is to show the proportion of codes that were 
covered separately by domain to support inferences about overall 
programs. Programs that cover a large proportion of the codes could be 
prioritized for dissemination, and the related content from such pro- 
grams could be used to supplement other programs (e.g., combining the 
best aspects of various programs). 

Future studies are needed to continue to refine and expand this 
coding methodology to apply across health promotion programs. This 
tool and methodology appear promising for identifying programs to 
prioritize for dissemination, guiding the selection of programs or pro- 
gram content to fit the needs of individual contexts, and identifying 
gaps across programs to assist efforts to develop and test supplemental 
implementation supports. 


4.1. Strengths and limitations 


A strength of this work was the use of the CFIR, which is a com- 
prehensive and commonly-used implementation science framework 
that has well-developed publicly available resources. One limitation of 
this study is that despite creating a 3-level scale to capture variability in 
the extent of meeting each code, few programs received a score of 
“more than minimally developed” on any of the codes. This may be due 
to the rarity of programs including exceptional implementation re- 
sources; however, another possibility is that the thresholds were poorly 
defined. Future studies should provide detailed definitions for coders to 
use in differentiating minimal versus extensive resources. Another 
limitation was that end user and stakeholder input was not directly 
solicited, other than reviewing what was covered in scientific publica- 
tions. Authors did, however, communicate study findings with program 
sponsors (when contact information was available), in order to provide 
feedback that may be relevant to program revisions. Future studies 
should consider incorporating a participatory approach with end users 
to obtain perspectives about what users are seeking in health promotion 
programs and resources, although this likely varies by topic and con- 
tent. 


4.2. Conclusions 


Implementation factors are critical determinants of implementation 
success (Tomoaia-Cotisel et al., 2013; Damschroder et al., 2009). One 
next step in supporting more widespread implementation and effec- 
tiveness of setting- and organization-based health promotion programs 
should be prioritizing the dissemination of programs that address the- 
oretically-based implementation factors, and that supplement already- 
developed programs with implementation supports. This study de- 
scribes a tool and methodology for assessing implementation factors 
within existing programs, which can be adapted for other health pro- 
motion programs to support their success and impact. Such tools and 
methodologies can be used by researchers and stakeholders to better 


C.M. Bejarano, et al. 


support practitioners and other end users to engage in successful im- 
plementation efforts. Practitioners can utilize this information to guide 
the selection of programs based on their individual needs and com- 
prehensiveness of the program. A better consideration of real-world 
implementation factors in the development and dissemination of health 
promotion programs is likely needed for such programs to have an in- 
creased impact on public health. 

Supplementary data to this article can be found online at https:// 
doi.org/10.1016/j.pmedr.2019.100909. 
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